Assessment of Data Quality of Selected Data Sets in the Department of Energy/comprehensive Nuclear Test-ban- Treaty Knowledge Base

The U.S. Department of Energy’s (DOE) Comprehensive Nuclear Test-Ban-Treaty (CTBT) Knowledge Base (KB) contains detailed regional data, from which corrections to seismic, hydroacoustic, and infrasonic signals will be generated. As the KB is populated with information and data sets detailing regional geological and geophysical structures and reference event data, questions of “How good is the information?” and “What confidence can I have in the corrections?” arise. This report documents work todate at the Pacific Northwest National Laboratory on the development of a “toolbox” of statistically-based algorithms which may be used to assess the quality of individual data sets, and consistency across multiple data sets, both on data in the KB and prior to including new data in the KB. Thirteen data sets (consisting of metadata, header, projection, and data files), supplied by Sandia National Laboratories, from the KB were used in this preliminary examination. The metadata files were reviewed before analysis of the data sets began. We noted that some fields were not filled in, others had very brief entries, while yet others were quite complete and informative. Comparing metadata files across data sets, we noted that the quality of the information was not consistent, there were problems with the accuracy/ precision of the numerical data, processing audit trails were poor to nonexistent, and when compared to the headers in the data files, some discrepancies were noted. Several methods were employed to evaluate the individual data sets for spurious data. We believe that because these gridded data sets in the KB are composites created from multiple sources and have been processed and smoothed, no outlier data was found. We did discover in some of the data sets that areas of constant value (algorithm default values) existed, which are not geologically reasonable. We believe that these areas were created as a result of the processing and are not valid data. Problems with agreement between data sets were also identified. Comparing data sets was problematic because of the different grid sizes and cell locations. As an example of what can be done to evaluate agreement between data sets, we examined three Mohorovicic (Moho) Discontinuity depth maps, which contained regions in common to two or all three data sets. Depths to, and trends in, the Moho in the data sets did not agree with each other, and in some instances depths at the same location had differences of up to 20 km. These are all serious problems, which must be rectified prior to using this KB data to generate seismic corrections. The effects of the noted discrepancies on the corrections have not yet been assessed, but we believe that corrections derived from using the different data sets would be significantly different. The cumulative effects of multiple errors could be more drastic. Certain needs for the overall KB were also identified, such as a need for well-defined criteria for accuracy of data, estimates of uncertainty to be associated with individual data, a consistent schema for gridding and combining data sets, and tolerance limits for agreement between data sets. Treatment of uncertainty in the data and understanding the effects of that error on the correction estimates that result from using the KB data is essential. The data sets from the KB used in this study are not raw data. They have been generated from multiple sources, processed, interpolated, and smoothed, and have no estimates of error or uncertainty. Uncertainty estimates cannot be confidently derived from processed and smoothed data, as from raw data. Any estimates of error and uncertainty will need to be inferred from ground-truth events.